Imagination Machines: A New Challenge for Artificial Intelligence

The aim of this paper is to propose a new overarching challenge for AI: the design of imagination machines. Imagination has been defined as the capacity to mentally transcend time, place, and/or circumstance. Much of the success of AI currently comes from a revolution in data science, specifically the use of deep learning neural networks to extract structure from data. This paper argues for the development of a new field called imagination science, which extends data science beyond its current realm of learning probability distributions from samples. Numerous examples are given in the paper to illustrate that human achievements in the arts, literature, poetry, and science may lie beyond the realm of data science, because they require abilities that go beyond finding correlations: for example, generating samples from a novel probability distribution different from the one given during training; causal reasoning to uncover interpretable explanations; or analogical reasoning to generalize to novel situations (e.g., imagination in art, representing alien life in a distant galaxy, understanding a story about talking animals, or inventing representations to model the large-scale structure of the universe). We describe the key challenges in automating imagination, discuss connections between ongoing research and imagination, and outline why automation of imagination provides a powerful launching pad for transforming AI. “Imagination is more important than knowledge. Knowledge is limited. Imagination encircles the world.” – Albert Einstein ‘Imagine there’s no countries. It isn’t hard to do. Nothing to kill or die for. And no religion too” – Song by John Lennon Artificial intelligence is poised to become the “electricity” of our age (Ng 2016), transforming industries across a wide spectrum of areas, from autonomous driving to voiceactivated virtual personal assistants. However, these successes of AI, powered by data science (Murphy 2013) and deep learning (Goodfellow, Bengio, and Courville 2016), may not be sufficient for AI to be capable of matching human capabilities in the long run. This paper focuses specifically on one core capability – imagination – and discusses why its automation may be fundamental to the continuing success of AI in the coming decades. Copyright c © 2018, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. Figure 1: Jean-Michel Basquiat’s untitled painting of a human skull sold recently at a New York auction for over 100 million dollars. Art is a paradigmatic example of the imaginative capacity of humans. The Oxford Handbook of the Development of Imagination defines imagination as the capacity to mentally transcend time, place, and/or circumstance (Taylor 2013). Einstein prized imagination because it enabled him to pose hypothetical questions, such as “What would the world look like if I rode a beam of light”, a question that led him to develop the revolutionary theory of special (and later, general) relativity. Imagination is a hallmark of counterfactual and causal reasoning (Pearl 2009). Imagination also provides the foundational basis for art (see Figure 1). Basquiat’s painting illustrates what is special about imagination in art: fidelity to the original is not the objective here, but rather the striking use of colors and textures to signify an illusion. In John Lennon’s famous song “Imagine”, he asks us to contemplate a world without countries, an abstraction of reality that is a hallmark of imaginative thinking. In Beethoven’s Pastoral symphony, each of the five movements portrays a particular aspect of nature, from the slow movement depicting the motion of a stream to the strenuous fourth movement depicting the arrival of a storm and thunder. Imagination plays a central role in the lives of children and adults. The runaway success of the Harry Potter series shows what a gifted writer can accomplish in holding the attention of children, highlighting the crucial role that make-believe plays in the formative years of children. Wonder Woman was the smash $1 billion Hollywood hit of the year, showing once again that the world of fantasy and imagination is one sure fire way to create a money making movie. Although imagination has attracted the attention of some researchers, the early work on this topic has been somewhat limited in scope (Alexander 2001), and more recent work has explored this topic in rather restricted situations (Pascanu et al. 2017; Elgamman et al. 2017). This brief paper summarizes several converging lines of argument as to why imagination machines constitutes a broad comprehensive research program that has the potential to transform AI in the next few decades. Imagination is one of the hallmarks of human intelligence (Asma 2017), an ability that manifests itself in children at a very young age, and prized by society in many endeavors, from art (see Figure 1) and literature to science. It represents an area largely ignored by most AI research, although tantalizing glimpses of the power of imagination are beginning to manifest themselves in different strands of current AI research, as will be discussed below. As work by the Nobel-prize winning economist Daniel Kahneman (with his late colleague, Amos Tversky) has shown, based on many empirical studies, human decision making does not conform to the maxims of expected utility theory. Faced with a “lottery” (a decision problem with several uncertain outcomes with different payoffs), human decision making often does not result in picking choices that have the maximum expected utility. Year after year, in state after state, millions of Americans buy lottery tickets, because they can “imagine” themselves winning and becoming rich, despite the vanishingly small probability of winning. Clearly, for many humans, imagination in this case (mis)guides their actions into violating the principle of maximizing expected utility. From Data Science to Imagination Science “A theory is not like an airline or bus timetable. We are not interested simply in the accuracy of its predictions. A theory also serves as a base for thinking. It helps us to understand what is going on by enabling us to organize our thoughts. Faced with a choice between a theory which predicts well but gives us little insight into how the system works and one which gives us this insight but predicts badly, I would choose the latter, and I am inclined to think that most economists would do the same.” – Ronald Coase, Nobel-prize winning economist. “I now take causal relationships to be the fundamental building blocks both of physical reality and of human understanding of that reality, and I regard probabilistic relationships as but the surface phenomena of the causal machinery that underlies and propels our understanding of the world”. – Judea Pearl, Causality. The ability to coax structure out of large datasets, particFigure 2: Generative Adversarial Networks (GANs) (Goodfellow et al. 2014) can create images from samples of a fixed distribution, but imaginative art such as Basquiat’s painting in Figure 1 require going beyond reproducing existing art. A variant of a GAN, called a “creative adversarial network” attempts to generate “novel” art (Elgamman et al. 2017), producing the images shown above. ularly for difficult to program tasks, such as computer vision, speech recognition, and high-performance game playing, has led to significant successes of machine learning in a variety of real-world tasks, particularly using deep learning approaches. Broadly speaking, machine learning or data science is the process of constructing a probability distribution from samples, or equivalently being able to generate new samples from given samples that fool an expert discriminator (Goodfellow et al. 2014). The fundamental difference between data science and imagination science is that the latter extends to realms far beyond the former: for example, imagination science addresses the problem of generating samples that are “novel”, meaning they come from a distribution different from the one used in training. Imagination science also addresses the problem of causal reasoning to uncover simple explanations for complex events, and uses analogical reasoning to understand novel situations. Can computers produce novel art like Basquiat’s painting? Recent work on a variant of a generative adversarial network called CAN (for Creative Adversarial Network) (see Figure 2) shows that computers can be trained to produce images that are both art, as well as differ from standard styles, like impressionism or cubism. While CANs are a useful step forward, building on Berlyne’s theory of novelty (Berlyne 1960), their architecture is currently specific to art, and not general enough to provide a computational framework for imagination. However, it does suggest one possible avenue to designing an Imagination Network architecture. Other extensions of GAN models, such as CycleGAN (Zhu et al. 2017), are suggestive, but such extensions are at present tailored to visual domains, and even in that circumscribed setting, only capable of specific generalizations (e.g., turning Monet styled watercolor paintings into what look like digital photographs of the original scene). Most machine learning is based on the discovery and exploitation of statistical correlations from data, including approaches using parametric graphical model representations (Murphy 2013) or kernel-based non-parametric representations (Schölkopf and Smola 2002), and most recently, nonlinear neural net based models (Goodfellow, Bengio, and Courville 2016). Correlation, as has been pointed out many times, is not causation, however, and causal reasoning is one of the primary hallmarks of human imaginative reasoning (Pearl 2009). One of the primary rationales for causal reasoning is the need to provide comprehensible explanations, which will become increasingly important as autonomous systems play an ever larger role in society. A self-driving car that gets involved in an accident may be required to provide an explanation of its behavior, much as a human driver would, and such explanations often take on a causal form (see Figure 3). A hallmark of imagination is the ability to reason about counterfactuals (Pearl 2009). The links between causal reasoning and imagination are explored from a probabilistic Bayesian perspective in (Walker and Gopnik 2013). Humans seek causal explanations because they want to understand the world in simple “cause-effect” relationships. They make analogies to interpret strange worlds, like the interior of an atom, in terms of worlds they understand, like the solar system, even though such analogies are imperfect. As Coase suggests above, humans desire interpretable explanations, even at the expense of fidelity to reality. Imaginative Perception: Labels to Affordances “Although the field of A.I. is exploding with microdiscoveries, progress toward the robustness and flexibility of human cognition remains elusive. Not long ago, for example, while sitting with me in a cafe, my 3-year-old daughter spontaneously realized that she could climb out of her chair in a new way: backward, by sliding through the gap between the back and the seat of the chair. My daughter had never seen anyone else disembark in quite this way; she invented it on her own and without the benefit of trial and error, or the need for terabytes of labeled data.” – Gary Marcus, Artificial Intelligence is Stuck: Here’s how to Move it forward., New York Times Sunday Review, July 29, 2017. Alison Gopnik, a well known psychologist, in a recent article in Scientific American titled “Making AI More Human”, marveled at the fact that “my five-year-old grandson, Augie, has learned about plants, animals and clocks, not to mention dinosaurs and spaceships. He also can figure out what other people want and how they think and feel. He can use that knowledge to classify what he sees and hears and make new predictions”(Gopnik 2017). One of the successes of machine learning, specifically deep neural networks (Goodfellow, Bengio, and Courville 2016), is object recognition. Performance on certain fixed datasets, such as Imagenet, has been steadily improving (Krizhevsky, Sutskever, and Hinton 2017). Under certain specific conditions, where large amounts of labeled datasets are available for narrowly defined tasks, deep learning approaches are able to exceed human level performance, a remarkable achievement. However, these results have to be interpreted with caution. There are as yet no well-defined procedures for extracting interpretable explanations from deep learning networks, and innocuous amounts of imperceptible noise appear sufficient to make a deep learning network guess the wrong label (Nguyen, Yosinski, and Clune 2015). Children, such as Augie, generally do not excel at highperformance problem-solving in narrowly constrained problems, be it Imagenet or Go (Silver et al. 2016) or Jeopardy (Tesauro et al. 2014), but rather demonstrate extremely versatile competence at comprehending the world in all its multi-modal richness and dimensionality. AI researchers focusing almost entirely at superhuman performance on artificial datasets or puzzles are at risk of losing sight of what makes humans like Augie truly special. Challenge tasks in computer vision, speech recognition, and other areas, focus on the ability to label a particular object or scene (or transcribe a given dialog), where the emphasis is on expert level ability given a statically defined task. Children, in contrast, are capable of learning in a much more fluid manner, coping with significant variability between training and test distributions, and they seem to be able to learn quite effectively without requiring much explicit labeling. Recent work is beginning to address the importance of imaginative causal reasoning in enabling neural net approaches to learn more effectively without labels (Stewart and Ermon 2017). However, a child interprets objects with an imaginative flexibility that lies far beyond what any AI system can accomplish today. To a child, a chair may serve as a hiding place, by crouching under it, or a stool, to retrieve another object placed beyond reach on a high table. In other words, using the terminology introduced by the psychologist James Gibson, imaginative thinking in perception revolves centrally around the core concept of “affordances”: an object is perceived in terms of the actions it enables an agent to do, and not purely in terms of a descriptive label (Gibson 1977). A bed may suggest lying down to an adult, but to a child, it means many different things, including the ability to jump up and down on it, as children are apt to do. Affordances are also essential to the design of everyday appliances (Norman 2002). What would it take to develop “Imagination Networks”, an imaginative perceptual system that is able to interpret images with the same flexibility and richness of behavior that Gary Marcus’ 3 year old child demonstrates, or the breadth of knowledge of Alison Gopnik’s five-year-old grandson, Augie? For one, the ability to recognize and exploit affordances. Second, the ability to integrate perceptual information into an agent’s goals, which are entirely a function of the agent’s body. Affordances, like being able to get in and out of small openings in the back of a chair, depend on an agent’s physical size and its capabilities. Simulated agents that function in video games, such as Atari, may have affordances that depend on their particular capabilities. Imaginative perception also plays a key role in other perceptual abilities, such as interpreting speech intonations and emotions, as well as body gestures. Affordances play a central role (a) ALVINN learned to drive from human supervised data. (b) Imagination in ALVINN Figure 3: ALVINN (Pomerleau 1989) was an early attempt at building a self-driving vehicle that learned from observing human driving behavior. To accelerate learning, ALVINN employed a simple causal model of driving to imagine many hypothetical driving situations from each real experience. in connecting objects with the actions they enable agents to undertake. The computation of affordances is an important objective for extending current work in deep learning for computer vision. Work on affordance recognition is still in its infancy, but important steps have been taken (Rome, Hertzberg, and Dorffner 2008). We discuss below how affordances can be learned by recognizing topological features in environments. Imagination in Problem Creation “Perhaps no single phenomenon reflects the positive potential of human nature as much as intrinsic motivation, the inherent tendency to seek out novelty and challenges, to extend and exercise one’s capacities, to explore, and to learn. Developmentalists acknowledge that from the time of birth, children, in their healthiest states, are active, inquisitive, curious, and playful, even in the absence of specific rewards” (Ryan and Deci 2000). Much AI research has been focused on problem solving, but imagination provides the ability to do problem creation. A landmark early example was Doug Lenat’s AM system (Lenat and Brown 1983), which was able to conjecture a large number of interesting theorems in elementary number theory (but lacked the ability to prove any of them). We now have many sophisticated ways to solve large Markov decision processes (Sutton and Barto 1998), but we lack the knowhow to create new MDPs. Deep reinforcement learning agents (Mnih et al. 2015) can play a precisely formulated Atari video game endlessly without getting bored, but the ability to create a new Atari video game remains completely out of reach of these systems. Yet, game design is a hallmark of imagination in human game developers, a skill whose success can be measured in billions of dollars of revenue. It has long been recognized that reinforcement learning systems require the human to supply a reward function of the task. Yet, children seem capable of learning a wide variety of tasks, seemingly without explicit reward functions being supplied. One possible way humans acquire rewards is to learn them from observing the behavior of other humans, an approach referred to as inverse reinforcement learning (Ng and Russell 2000). Another hallmark of imagination is the ability to get curious, to seek out novel situations, and to get bored solving the same problem repeatedly. Increasingly, many CEOs and managers have recognized is that excellence is often a by product of giving humans more autonomy in their work (Pink 2009). Intrinsic motivation has been studied in psychology for several decades (Ryan and Deci 2000), and now receiving increasing attention in machine learning (Singh, Barto, and Chentanez 2004). Imagination in Language: Metaphors “I done wrestled with an alligator, I done tussled with a whale; handcuffed lightning, thrown thunder in jail; only last week, I murdered a rock, injured a stone, hospitalized a brick; I’m so mean I make medicine sick.” – Muhammad Ali. Metaphors play a crucial role in language (Lakoff and Johnson 1980), where phrases like ”The stock market crashed” are apt to be used in everyday language. The creative use of metaphors in language, such as the above quotation from Muhammad Ali, shows the power of imagination in language. Recent successes in natural language processing, such as machine translation, build on deep learning sequence learning models, such as long short-term memory (LSTM) (Gers et al. 2002). However, the ability to undertake routine translation is far removed from the ability to creatively use language. Recently, several techniques have emerged for mapping words into vectors, like word2vec (Mikolov, Yih, and Zweig 2013) or GLOVE (Pennington, Socher, and Manning 2014). Such word embedding systems can be trained on a large corpus of words, like Wikipedia, and produce continuous representations, which can be used to reason about linguistic relations (such as “Man is to Woman as King is to X”). A significant challenge remains in showing how word embedding techniques can be extended so that they can provide the basis for generating new metaphors or richly descriptive phrases of the sort quoted above, which lie at the heart of imaginative language use. What insights will enable AI to generate imaginative poetry? ”Beauty is truth, truth beauty, – that is all Ye know on earth, and all ye need to know.” Ode on a Grecian Urn, John Keats. Probabilistic Imaginative Models Thus far, the paper has discussed the problem of designing imagination machines in a non-technical manner. In this last section, we briefly sketch out some ideas for how to develop a more rigorous mathematically-based theory of imagination. Machine learning is based on the core concept of a probabilistic generative model (PGM) (Murphy 2013), which concisely summarizes the distribution that generates both the training as well as the test datasets. Examples of PGMs include Gaussian mixture models, hidden Markov models, and Bayesian networks. Building on this concept, (a) PGM Dt (unlabeled). The aim of domain adaptation, as reviewed above, is to construct an intermediate representation that combines some of the features of both the source and target domains, with the rationale being that the distribution of target features differs from that of the source. Relying purely on either the source or the target features is therefore suboptimal, and the challenge is to determine what intermediate representation will provide the best or optimal transfer between the two domains. In the CORAL objective function in Equation 1, the goal was to find a transformation A that makes the source covariance resemble the target as closely as possible. Our approach simplifies Equation 1 by restricting the transformation matrix A to be a SPD matrix, and furthermore, we solve the resulting equation exactly on the manifold of SPD matrices. More formally, we solve the following Riccati equation (Bhatia 2007):